{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:100% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to EvoXBench\n",
    "\n",
    "In this notebook, we will demonstrate \n",
    "- how to install EvoXBench\n",
    "- the basics of EvoXBench\n",
    "\n",
    "**[EvoXBench](https://arxiv.org/abs/2208.04321)** is an efficient platform \n",
    "for facilitating neural architecture search (NAS) \n",
    "without the requirement of *GPUs* or \n",
    "sophisticated deep learning packages, such as *PyTorch, TensorFlow*, etc.\n",
    "\n",
    "![](https://raw.githubusercontent.com/EMI-Group/evoxbench/main/assets/evoxbench_overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparation \n",
    "Let's perform the following steps to have EvoXBench properly installed. \n",
    "\n",
    "First, download the following two files:\n",
    "- ``database_xxx.zip`` from [Google Drive](https://drive.google.com/file/d/11bQ1paHEWHDnnTPtxs2OyVY_Re-38DiO/view?usp=sharing) or [Baidu NetDisk](https://pan.baidu.com/s/1PwWloA543-81O-GFkA7GKg)\n",
    "- ``data_xxx.zip`` from [Google Drive](https://drive.google.com/file/d/1fUZtpTjfEQao2unLKaspL8fOq4xdSXt2/view?usp=sharing) or [Baidu NetDisk](https://pan.baidu.com/s/1yopkISKyjbWIHXFV_Op3pg)\n",
    "\n",
    "Second, unzip these two files and find their paths\n",
    "- my ``database`` and ``data`` are unzipped to:\n",
    "```python\n",
    "    # /Users/luzhicha/Dropbox/2023/github/evoxbench/\n",
    "    # └─ database/\n",
    "    # |  |  __init__.py\n",
    "    # |  |  db.sqlite3\n",
    "    # |  |  ...\n",
    "    # |  \n",
    "    # └─ data/\n",
    "    #    └─ darts/\n",
    "    #    └─ mnv3/\n",
    "    #    └─ ...\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing EvoXBench...\n"
     ]
    }
   ],
   "source": [
    "print('Installing EvoXBench...')\n",
    "! pip install evoxbench 1>/dev/null\n",
    "# ! pip install git+https://github.com/EMI-Group/evoxbench "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configurating EvoXBench...\n",
      "Configuration Succeed!\n"
     ]
    }
   ],
   "source": [
    "print('Configurating EvoXBench...')\n",
    "from evoxbench.database.init import config\n",
    "# make sure you update these two paths accordingly, and the first path should be for database file\n",
    "config(\"/Users/luzhicha/Dropbox/2023/github/evoxbench/database\",\n",
    "       \"/Users/luzhicha/Dropbox/2023/github/evoxbench/data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good! Now we have successfully installed and configured **EvoXBench**. Let's now get started with some quick examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.1 How to create a NAS benchmark (search space)\n",
    "\n",
    "**EvoXBench** currently supports the following seven search spaces\n",
    "\n",
    "| $\\Omega$ | $D$ | $|\\Omega|$ | Objectives | Dataset |\n",
    "|:-:|:-:|:-:|:-:|:-:|\n",
    "| [NB101](https://github.com/google-research/nasbench) | 26 |423K | $f^{e}$, ${f}^{c}$ | CIFAR-10 |\n",
    "| [NB201](https://github.com/D-X-Y/NAS-Bench-201) | 6 | 15.6K | $f^{e}$, ${f}^{c}$, ${f}^{\\mathcal{H}}$ | CIFAR-10 |\n",
    "| [NATS](https://github.com/D-X-Y/NATS-Bench) | 5 | 32.8K | $f^{e}$, ${f}^{c}$, ${f}^{\\mathcal{H}}$ | CIFAR-10 |\n",
    "| [DARTS](https://github.com/automl/nasbench301) | 32 | $\\sim10^{21}$ | $f^{e}$, ${f}^{c}$ | CIFAR-10 |\n",
    "| [ResNet-50](https://github.com/mit-han-lab/once-for-all) | 25 | $\\sim10^{14}$ | $f^{e}$, ${f}^{c}$ | ImageNet-1K |\n",
    "| [Transformer](https://github.com/microsoft/Cream/tree/main/AutoFormer) | 34 | $\\sim10^{14}$ | $f^{e}$, ${f}^{c}$ | ImageNet-1K |\n",
    "| [MNV3](https://github.com/mit-han-lab/once-for-all) | 21 | $\\sim10^{20}$ | $f^{e}$, ${f}^{c}$, ${f}^{\\mathcal{H}}$ | ImageNet-1K |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Benchmaking on NB101 search space with objectives: err&params\n"
     ]
    }
   ],
   "source": [
    "# NAS-Bench-101 search space\n",
    "from evoxbench.benchmarks import NASBench101Benchmark\n",
    "objs = 'err&params'  # ['err&params', 'err&flops', 'err&params&flops']\n",
    "benchmark = NASBench101Benchmark(objs=objs, normalized_objectives=False)\n",
    "print(\"Benchmaking on NB101 search space with objectives: {}\".format(objs))\n",
    "\n",
    "# # NAS-Bench-201 search space\n",
    "# from evoxbench.benchmarks import NASBench201Benchmark\n",
    "# # hardware = 'edgegpu'  # ['edgegpu', 'raspi4', 'edgetpu', 'pixel3', 'eyeriss', 'fpga']\n",
    "# # ['err&params', 'err&flops', 'err&latency', 'err&params&flops', 'err&params&latency', ...]\n",
    "# objs = 'err&params&flops&edgegpu_latency&edgegpu_energy'\n",
    "# benchmark = NASBench201Benchmark(objs=objs, normalized_objectives=False)\n",
    "# print(\"Benchmaking on NB201 search space with objectives: {}\".format(objs))\n",
    "\n",
    "# # NATS size search space \n",
    "# from evoxbench.benchmarks import NATSBenchmark\n",
    "# objs = 'err&params&flops&latency'\n",
    "# # ['err&params', 'err&flops', 'err&latency', 'err&params&flops', 'err&params&latency', ...]\n",
    "# benchmark = NATSBenchmark(objs=objs, normalized_objectives=False)\n",
    "# print(\"Benchmaking on NATS search space with objectives: {}\".format(objs))\n",
    "\n",
    "# # DARTS search space\n",
    "# from evoxbench.benchmarks import DARTSBenchmark\n",
    "# objs = 'err&params'  # ['err&params', 'err&flops', 'err&params&flops']\n",
    "# benchmark = DARTSBenchmark(objs=objs, normalized_objectives=False)\n",
    "# print(\"Benchmaking on DARTS search space with objectives: {}\".format(objs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.2 How to evaluate an architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Randomly create 1 architectures:\n",
      "[{'matrix': array([[0, 1, 0, 0, 0, 1, 1],\n",
      "       [0, 0, 1, 1, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 1, 0, 1],\n",
      "       [0, 0, 0, 0, 1, 1, 0],\n",
      "       [0, 0, 0, 0, 0, 1, 1],\n",
      "       [0, 0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0, 0]]), 'ops': ['input', 'conv3x3-bn-relu', 'conv3x3-bn-relu', 'conv1x1-bn-relu', 'conv3x3-bn-relu', 'conv3x3-bn-relu', 'output']}]\n"
     ]
    }
   ],
   "source": [
    "# let's randomly create N architectures\n",
    "N = 1\n",
    "archs = benchmark.search_space.sample(N)\n",
    "print('Randomly create {} architectures:'.format(N))\n",
    "print(archs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encode architectures to decision variables X: \n",
      "[[1 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 1 1 0 0 0 1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# encode architecture (phenotype) to decision variables (genotypes)\n",
    "X = benchmark.search_space.encode(archs)\n",
    "print('Encode architectures to decision variables X: ')\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating architectures for objectives: err&params\n",
      "[[6.95446134e-02 8.55553000e+06]]\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the objective values\n",
    "# if true_eval is True, return mean TEST accuracy over multiple runs, \n",
    "# should only be used for final comparison.\n",
    "true_eval = True\n",
    "F = benchmark.evaluate(X, true_eval=true_eval)\n",
    "print(\"Evaluating architectures for objectives: {}\".format(objs))\n",
    "print(F)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating architectures for objectives: err&params\n",
      "Trial 1:\n",
      "[[1.05033886e-01 1.00279400e+06]]\n",
      "Trial 2:\n",
      "[[8.94077586e-02 1.00279400e+06]]\n",
      "Trial 3:\n",
      "[[1.0496355e-01 1.0027940e+06]]\n",
      "Trial 4:\n",
      "[[9.64054196e-02 1.00279400e+06]]\n",
      "Trial 5:\n",
      "[[1.0496355e-01 1.0027940e+06]]\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the objective values\n",
    "# if true_eval is False, return VALIDATION accuracy from one (randomly selected) run, \n",
    "# should be used during search\n",
    "true_eval = False\n",
    "print(\"Evaluating architectures for objectives: {}\".format(objs))\n",
    "for i in range(5):\n",
    "    F = benchmark.evaluate(X, true_eval=true_eval)\n",
    "    print(\"Trial {}:\".format(i+1))\n",
    "    print(F)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
